Assignment Day 11 | July 20¶
Question:¶
Find the correlation of Attrition variable with all other variables in the dataset
Problem Statement:¶
A large company named XYZ, employs, at any given point of time, around 4000 employees. However, every year, around 15% of its employees leave the company and need to be replaced with the talent pool available in the job market. The management believes that this level of attrition (employees leaving, either on their own or because they got fired) is bad for the company, because of the following reasons -
The former employees projects get delayed, which makes it difficult to meet timelines, resulting in a reputation loss among consumers and partners A sizeable department has to be maintained, for the purposes of recruiting new talent More often than not, the new employees have to be trained for the job and/or given time to acclimatise themselves to the company
Hence, the management has contracted an HR analytics firm to understand what factors they should focus on, in order to curb attrition. In other words, they want to know what changes they should make to their workplace, in order to get most of their employees to stay. Also, they want to know which of these variables is most important and needs to be addressed right away.
Since you are one of the star analysts at the firm, this project has been given to you.
Goal of the case study You are required to model the probability of attrition. The results thus obtained will be used by the management to understand what changes they should make to their workplace, in order to get most of their employees to stay.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pandas_profiling import ProfileReport
import itertools
import scipy.stats as stats
from scipy.stats import ttest_1samp, ttest_ind,mannwhitneyu,levene,shapiro,wilcoxon
from statsmodels.stats.power import ttest_power
from scipy.stats import linregress
import warnings
warnings.filterwarnings('ignore')
# to visualise al the columns in the dataframe
pd.pandas.set_option('display.max_columns', None)
df = pd.read_csv('general_data.csv')
map_attrition = {'Yes': 1, 'No': 0} # Convert the categorical object into numerical
map_gender = {'Male': 1, 'Female': 0}
df = df.replace({'Attrition': map_attrition})
df = df.replace({'Gender': map_gender})
df.head()
df.drop(labels=['EmployeeID','EmployeeCount','StandardHours','Over18'],axis = 1,inplace=True)
df.info()
pfr = ProfileReport(df, title="Attrition EDA")
pfr.to_file(output_file="Attrition Data Profiling.html")
pfr
df.describe().T
# Let us look at the target column which is 'Attrition' to understand how the data is distributed amongst the various values
df.groupby(["Attrition"]).count()
sns.countplot(x='Attrition', data=df)
#missing data => Find total rows having missing values and calculate the percentage of missing values for each field.
total = df.isnull().sum().sort_values(ascending=False)
percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)
missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
missing_data.head(20)
#Let us drop the na values as they are minimal here
df.dropna(inplace=True)
#missing data => Find total rows having missing values and calculate the percentage of missing values for each field.
total = df.isnull().sum().sort_values(ascending=False)
percent = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)
missing_data = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
missing_data.head(20)
vars_num = list(df.select_dtypes([np.number]).columns) # List of Numerical Columns
vars_num
fig = plt.figure()
for var in vars_num:
if (var != 'Attrition' and var != 'Gender'):
fig = plt.figure(figsize =(5, 5))
sns.boxplot(y=df[var],orient="v")
df.boxplot(rot=90,figsize=(15,15),fontsize='large')
df.hist(figsize=(15,15))
cols = vars_num
length = len(cols)
cs = ["b","r","g","c","m","k","lime","c","olive"]
fig = plt.figure(figsize=(20,20))
for i,j,k in itertools.zip_longest(cols,range(length),cs):
if (i != 'Attrition' and i != 'Gender'):
plt.subplot(4,4,j+1)
ax = sns.distplot(df[i],color=k,rug=True)
ax.set_facecolor("w")
plt.axvline(df[i].mean(),linestyle="dashed",label="mean",color="k")
plt.legend(loc="best")
plt.title(i,color="navy")
plt.xlabel("")
# Let us do a bivariate analysis with pair plot
sns.pairplot(df[vars_num],diag_kind='auto')
corr_mat = df[vars_num].corr()
corr_mat
plt.figure(figsize=(12,10))
with sns.axes_style("darkgrid"):
sns.heatmap(corr_mat,cmap='magma', annot=True,fmt='.2f')
plt.title("Correlation between variables")
plt.show()
Attrition With Age
Null Hypothesis H0 -> There is no significant difference in people's Age with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's Age with the people either with Attrition Yes or Attrition as No
LinregressResult = linregress(df[df['Attrition'] == 1]['Age'].sample(n=700),df[df['Attrition'] == 0]['Age'].sample(n=700))
print(LinregressResult)
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['Age'].sample(n=700),
df[df['Attrition'] == 0]['Age'].sample(n=700))
#define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
Attrition With Distance From Home
Null Hypothesis H0 -> There is no significant difference in people's Distance from their Home with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's Distance from their Home with the people either with Attrition Yes or Attrition as No
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['DistanceFromHome'].sample(n=700)
,df[df['Attrition'] == 0]['DistanceFromHome'].sample(n=700)) #define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
Attrition With MonthlyIncome
Null Hypothesis H0 -> There is no significant difference in people's MonthlyIncome with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's MonthlyIncome with the people either with Attrition Yes or Attrition as No
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['MonthlyIncome'].sample(n=700),
df[df['Attrition'] == 0]['MonthlyIncome'].sample(n=700)) #define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
Attrition With PercentSalaryHike
Null Hypothesis H0 -> There is no significant difference in people's PercentSalaryHike with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's PercentSalaryHike with the people either with Attrition Yes or Attrition as No
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['PercentSalaryHike'].sample(n=700),
df[df['Attrition'] == 0]['PercentSalaryHike'].sample(n=700)) #define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
Attrition With YearsAtCompany
Null Hypothesis H0 -> There is no significant difference in people's YearsAtCompany with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's YearsAtCompany with the people either with Attrition Yes or Attrition as No
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['YearsAtCompany'].sample(n=700),
df[df['Attrition'] == 0]['YearsAtCompany'].sample(n=700)) #define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
Attrition With YearsSinceLastPromotion
Null Hypothesis H0 -> There is no significant difference in people's YearsSinceLastPromotion with the people either with Attrition Yes or Attrition as No
Alternate hypothesis Ha -> There is significant difference in people's YearsSinceLastPromotion with the people either with Attrition Yes or Attrition as No
pearson_coef, p_value = stats.pearsonr(df[df['Attrition'] == 1]['YearsSinceLastPromotion'].sample(n=700),
df[df['Attrition'] == 0]['YearsSinceLastPromotion'].sample(n=700)) #define the columns to perform calculations on
print("Pearson Correlation Coefficient: ", pearson_coef, "and a P-value of:", p_value) # Results
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
if (p_value > 0.05):
print ("We reject our Null hypothesis and proceed with our Alternate hypothesis")
else:
print ("We accept our Null Hypothesis")
df[df.columns[0:]].corr()['Attrition'][:]
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['Age'].sample(n=700),
df[df['Attrition'] == 0]['Age'].sample(n=700),
data=df,kind="reg",color='b').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot('Attrition','MonthlyIncome',data=df,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['YearsAtCompany'].sample(n=700),
df[df['Attrition'] == 0]['YearsAtCompany'].sample(n=700),data=df,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['MonthlyIncome'].sample(n=700),
df[df['Attrition'] == 0]['MonthlyIncome'].sample(n=700)
,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot('Gender','MonthlyIncome',data=df,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot('YearsAtCompany','YearsWithCurrManager',data=df,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['YearsWithCurrManager'].sample(n=700),
df[df['Attrition'] == 0]['YearsWithCurrManager'].sample(n=700),data=df,kind="scatter",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['Gender'].sample(n=700),
df[df['Attrition'] == 0]['Gender'].sample(n=700),
data=df,kind="reg",color='r').annotate(stats.pearsonr)
plt.show()
sns.set(style="darkgrid", color_codes=True)
sns.jointplot(df[df['Attrition'] == 1]['YearsSinceLastPromotion'].sample(n=700),
df[df['Attrition'] == 0]['YearsSinceLastPromotion'].sample(n=700),
data=df,kind="reg",color='r').annotate(stats.pearsonr)
plt.show()
